Scalability and Production

When put into production code gets used more and on more data.

We will likely have to consider scalability of our methods in

• Computation time

• Memory requirements

When doing so we have to balance a trade-off between development costs and usage costs.

Example: Bayesian Inference

• MCMC originally takes ~24 hours

• Identifying and amending bottlenecks in code reduced this to ~24 minutes.

Is this actually better?

Knowing when to worry

This Lecture

Profiling your code: basics

t_start <- Sys.time()
Sys.sleep(0.5) # YOUR CODE
t_end <- Sys.time()

t_end - t_start

Time difference of 0.5097461 secs

library(tictoc)

tic() 
Sys.sleep(0.5) # YOUR CODE 
toc()

0.51 sec elapsed

tic("total")
tic("first, easy part")
Sys.sleep(0.5)
toc(log = TRUE)
## first, easy part: 0.509 sec elapsed
tic("second, hard part")
Sys.sleep(3)
toc(log = TRUE)
## second, hard part: 3.009 sec elapsed
toc()
## total: 3.523 sec elapsed

Profiling your code in detail

To diagnose scaling issues you have to understand what your code is doing.

• Stop the code at time \(\tau\) and examine the call-stack.

  • The current function being evaluated, the function that called that, the function that called that, …, top level function.

• Do this a lot and you can measure (estimate) the proportion of working memory (RAM) uses over time and the time spent evaluating each function.

Profiling: How To

source("assets/prof-vis-example.R")
profvis::profvis(f())

Notes on Time Profiling

• Will get slightly different results each time you run the function

  • Changes to internal state of computer
  • Usually not a big deal, mainly effects fastest parts of code
  • Be careful with stochastic simulations
  • Use set.seed() to make a fair comparison over many runs.

Notes on Profiling

pad_with_NAs

function(x, n_left, n_right){
  c(rep(NA, n_left), x, rep(NA, n_right))
}

mean

function (x, ...) 
UseMethod("mean")
<bytecode: 0x7f9aeaaea568>
<environment: namespace:base>

Memory Profiling

profvis() can similarly measure the memory usage of your code.

x <- integer()
for (i in 1:1e4) {
  x <- c(x, i)
}